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A  TRAINING  STRATEGY  FOR  OPERATIONAL  FIELD  TEST  DATA  COLLECTORS 

Introduction 

This  research  note  describes  a  training  strategy  developed  and  x^sed  for 
video  data  reduction  personnel  prior  to  their  participation  in  the  line-of- 
Si^t-Forward-Heavy  (LDS-F-H)  Force  Develc^ment  Test  and  Experimentation 
(FDIE) ,  Riase  II.  Althom^  develc^jed  e^qiressly  for  IDS-F-H  FDTE  II,  the 
principles  involved  may  be  applied  to  the  training  of  video  data  reducers 
(VERs)  assigned  to  any  type  of  cperational  field  testing  and  to  the  training 
of  data  collectors  in  general.  The  training  strategy  is  conprised  of  three 
phases:  (1)  system  overview  training,  (2)  data  collection  process  overview 
training,  and  (3)  individualized,  job  specific  training. 

Need  for  Training  Data  Collectors 

Army  guidance,  the  scientific  method,  and  practical  experience  underscore 
the  need  to  provide  afprcpriate  training  for  data  collectors  and  to  tailor  the 
training  to  the  data  collection  environment  (e.g. ,  field  or  laboratory) . 

Proper  instruction  of  these  personnel  facilitates  the  maintenance  of  a  high 
quality  data  collection  effort  by  ensuring  that  the  correct  observations  and 
measurements  are  made  with  the  required  precision.  Lack  of  appropriate 
training  increases  the  probability  that  the  data  will  be  inaccurate  and  thus 
lead  to  incorrect  conclusions. 

Data  collection  procedures  vhich  yield  inaccurate  findings  are  a 
particular  problem  for  the  military  vhen  these  data  are  used  to  make  critical 
decisions  regarding  system  acquisition.  The  Department  of  Army  (DA)  (1976) 
has  published  official  Arrty  guidance  for  testing  and  evaluation.  DA  Paitphlet 
71-3  states  that  an  effective  data  collection  plan  is  essential  to  the  success 
cf  operational  field  tests  and  that  successful  inplementation  of  this  plan 
mandates  adequate  training  for  personnel  conducting  the  data  collection 
effort.  Adequate  training  according  to  this  guidance  includes,  but  is  not 
limited  to,  instruction  on  the  use  of  instrumentation,  data  collection 
procedures,  and  data  reduction  and  analysis  procedures.  Furthermore,  to  be 
effective,  data  collectors  must  be  familiar  with  all  aspects  of  the  test,  to 
include:  the  test  purpose  and  socpe,  test  item  characteristics  and  data 
management  plans. 

Further  guidance  from  Haverland  (1988)  and  Meister  (1965)  suggests  that 
data  collectors  must  be  thorou^ily  acquainted  with  the  equipment  under  test. 
Meister  advocates  special  training  for  data  collectors  who  are  unfamiliar  with 
the  system.  He  asserts  that,  at  a  minimum,  the  data  collector  should  be  able 
to  name  with  certainty  eill  the  major  parts  of  the  system  and  subsystem  with 
which  they  are  dealing.  The  data  collectors  ^ould  be  able  to  state  the 
functions  performed  by  the  equipment  and  its  cperators  and  knew  the  order  in 
vhich  these  functions  are  to  occur. 

In  addition,  Meister  (1985)  states  that  everyone  involved  in  the  data 
collection  process,  including  the  system  ejqjerts,  should  receive  training  and 
practice  in  methods  of  recording  data.  Meister  (1985)  also  holds  that  data 
collectors  must  be  tau^t  to  recognize  and  record  the  events  vhich  are 
inportant  to  the  process.  He  advocates  practicing  vmtil  the  data  collectors 
are  ccmfortable  with  the  procedure.  The  selection  of  data  collectors  prior  to 
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training  is  critical  according  to  Meister  (1965) .  He  cautions  that  if  these 
personnel  are  not  carefully  chosen,  they  can  becxime  the  primary  source  of  data 
containination . 

More  specific  and,  for  this  paper,  itore  relevant  evidence  for  the  need  to 
effectively  train  data  collectors  has  emerged  from  the  iirplementation  of 
innovative  techniques  such  as  those  used  for  recording  operator  perfonnance  in 
the  confined  space  of  airplane  cockpits  and  armored  vehicles.  This  particular 
data  collection  technology  has  presented  a  variety  of  challenges  in  selecting 
euid  training  individoeils  to  extract  and  use  the  information  from  video  tapes. 
For  exarrple,  the  Enpirically  Validated  Task  Analysis  (EVTA)  technique  iu  a 
highly  sophisticated  procedxire  vAiich  is  capable  of  gathering  permanent  records 
of  operator  activities  and  internal  and  external  communications.  An  enpirical 
record  of  activity  times  is  generated  through  a  conrputer  software  program 
\4iich  builds,  manages,  and  analyzes  the  resulting  data  base  (Chaffer,  1990) . 
According  to  its  developer,  the  extraction  and  use  of  data  from  the  EVTA  audio 
and  video  records  requires  a  specially  selected  and  trained  team. 

The  Problem 

Ihe  scientific,  practical,  and  Anty  guidelines  for  the  selection  and 
braining  of  data  collection  personnel  became  very  pertinent  to  the  Fort  Bliss 
Field  Unit  of  the  Array  Research  Institute  (ARI)  early  in  the  summer  of  1989. 

At  that  time,  preparations  for  the  LDS-F-H  FDTE  II  at  Fort  Hunter-Liggctt,  CA 
were  in  the  final  stages.  FDTE  II  was  a  multi-Jnillion  dollar  test  of  a  system 
v^iich,  if  it  meets  its  various  procurement  criteria,  will  cost  billions  to 
produce  and  field.  A  Manpov/er  and  Personnel  Integration  (MANPRINT)  evaluation 
of  the  LDS-F-H  system  was  to  be  conducted  from  the  data  gathered  and  analyzed 
during  FDTE  II.  The  primary  data  source  for  this  evaluation  was  video  tape 
recordings  of  operator  performance.  The  results  of  the  evaluation  would  be 
used  by  the  Office  of  the  Deputy  Chief  of  Staff  for  Personnel  (ODCSPER)  as 
irput  to  the  official  MANPRINT  assessment.  A  summary  of  the  MANPRINT 
assessment  would  be  briefed  at  the  milestone  decision  review  (DA,  1987) . 
Clearly  then,  the  quality  of  the  data  gathered  during  the  test  could 
significantly  impact  the  future  of  the  system. 

Given  the  iirportance  of  the  role  of  the  data  collectors  in  the  system 
acquisition  process,  there  was  definitely  a  problem  with  the  exist ii^  IDS-F-H 
FDTE  II  data  collection  plan.  According  to  this  plan,  the  only  training 
prescribed  for  these  personnel  was  a  40-hour  system  overview  course  conducted 
by  the  prime  contractor.  While  this  course  provides  an  excellent  overview  of 
the  system  and  fulfills  one  of  the  training  requirements  set  forth  by  Meister 
(1985)  and  Haverland  (1988) ,  it  does  not  suffice  as  training  for  the 
extraction  and  use  of  human  performance  data  from  video  tape  recordings.  Ihe 
first  order  of  business  to  remedy  this  situation,  therefore,  became 
formulation  of  a  strategy  for  the  appropriate  training  of  the  LDS-F-H  FDTE  II 
VDRs.  Because  of  the  immediacy  of  the  FDTE  II,  the  need  for  quick  action,  not 
elegance,  dictated  most  of  the  plcinning  directed  toward  finding  a  solution  to 
the  training  dilemma. 
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The  Bigger  Problqn? 


Ihe  lack  of  appropriate  training  for  the  FDIE  II  VDRs  held  the  potential 
for  serious  raiaifications  for  the  DDS-F-H  system.  Entering  the  test  situation 
with  only  the  40-hour  system  overview  course  and  without  receiving  training  in 
accordance  with  the  guidelines  set  forth  by  the  Tunriy  and  other  experts  in  the 
testing  and  evaluation  community  could  have  led  to  disastrous  consequences. 

The  potential  repercussions  lead  one  to  ^jeculate  whether  the  training  problem 
encountered  for  this  particular  test  act  this  particular  point  in  time  for  this 
particular  system  is  unique  or  is  pervasive  within  the  testing  and  evaluation 
environment.  If  indeed  this  problem  is  not  unique  to  the  IDS-F-H  FDTE  II, 
then  the  training  strategy  to  be  described  may  be  of  some  use  in  framing  a 
solution  to  the  bigger  problem. 

Description  of  a  VDR  Training  Strategy 

An  initial  step  in  devising  a  training  strategy  was  to  identify  personnel 
vrtio  could  develop  and  execute  a  VDR  training  strategy.  Fortunately,  the 
leaders  of  two  key  IDS-F-H  data  collection  teams.  Performance  Video  Reduction 
(FM?)  and  Battle  Drill  Analysis  (BDA) ,  had  the  necessary  qualifications  and 
were  available.  Both  of  these  individuals  were  eager  for  the  opportunity  to 
teach  VCRs  because  each  had  worked  with  inproperly  trained  personnel  in  prior 
test  situations.  With  the  support  of  these  individuals  secured,  a  three  phase 
training  strategy  was  devised.  Ihase  One  consisted  of  the  prime  contractor's 
LDS-F-H  system  overview  training.  During  Phase  Two,  students  received  an 
overview  of  the  data  collection  process  used  during  operational  field  tests. 
Phase  Three  provided  individual  data  collectors  with  detailed  training  for 
their  specific  jobs. 

Phase  One:  IDS-F-H  System  Overview  Training 

Phase  One  consisted  of  the  40-hour  system  overview  course  conducted  by 
the  prime  contractor.  This  course  was  a  prerequisite  for  anyone  connected 
with  IDS-F-H  testing  because  it  provided  a  basic  understanding  of  system 
operation.  The  students  in  the  course  were  personnel  who  would  fill  a  variety 
of  roles  during  FDTE  II. 

Phgise  Two;  lOS-F-H  Data  Collection  Process  Overview  Trednino 

Phase  Two  of  the  training  was  designed  to  provide  students  with  an 
overview  of  the  operational  field  test  data  collection  process.  This  rtaining 
was  conducted  by  the  PVR  and  BDA  team  leaders  and  given  concurrently  with  the 
IDS-F-H  system  overview  course.  Concurrent  instruction  was  possible  because 
much  of  the  system  overview  training  was  conducted  in  the  IDS-F-H  fire  unit. 
Only  two  students  could  be  accommodated  per  fire  unit  and  no  more  than  two 
fire  units  were  available  at  any  given  time.  Consequently,  most  of  the  class 
had  a  considerable  amount  of  "down-time".  This  time  was  used  by  the  PVR  and 
BDA  team  leaders  to  conduct  training  at  two  different  learning  stations  vhere 
the  relationship  among  the  different  data  elements  was  taught.  The  impact  of 
individual  data  collection  efforts  on  the  rest  of  the  data  collection  process 
and  on  the  test  objectives  also  was  described  at  these  learning  stations. 
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Students  benefitted  by  having  the  DDS-F-H  system  overview  c»urse  and 
Phase  Two  of  the  VDR  training  conducted  concurrently.  Because  the  tactical 
fire  units  were  available  to  the  students  during  this  period,  they  were  able 
to  verify  system  information  taught  during  Phases  One  and  TVo,  as  well  as 
beccrne  thoroughly  familiar  with  the  equipment. 

At  Learning  Station  One,  the  instructor  expanded  on  the  information 
presented  in  the  Phase  One  system  overview  cx>urse  and  showed  the  students  hew 
that  information  related  to  the  various  data  collection  efforts..  Ihe  class 
•was  -tau^t  ■the  difference  between  "pure"  system  functions  (hardware  only)  and 
functions  which  required  •the  interface  of  soldier  and  machine.  The  group  was 
shown  •the  -various  system  conponents  (e.g.,  the  lights,  knobs,  switches,  and 
dials)  cissociated  wi-th  their  da-ta  collection  efforts.  Ihey  learned  to 
identify  the  data  sources  for  their  jobs  and  they  were  •tau(^t  vhich  specific 
da-ta  needed  -to  be  collected  -to  meet  test  chjectives. 

Upon  carpletion  of  each  segment  of  instruction  at  Learning  S-tation  One, 
students  proceeded  -to  Learning  Station  Two.  At  Station  Two,  they  were  shown 
hew  -the  operator  of  the  lOS-F-H  system  ccxild  have  an  influence  on  the  various 
system  functions  taught  at  Sta-tion  One.  The  studen-ts  saw  that  -these 
influences  could  have  ei-ther  a  positive  or  negative  iirpact  on  total  system 
performance.  They  learned  that  even  ■the  alienee  of  data  was  meaningful 
because  it  could  indicate  -that  the  operator  had  omitted  a  required  step  in  a 
sequence.  The  inportance  of  human  performance,  not  just  equipment 
performance,  was  enphasized. 

Ihe  -training  received  at  the  "two  learning  stations  also  provided  -the 
students  with  a  knowledge  of  instruTDentation  anomalies.  Ihis  knowledge 
prepared  them  to  conpare  information  gathered  from  different  sources  (e.g. , 
the  instrumented  fire  unit  or  field  test  environment  and  the  video  tape 
recordings)  and  to  reconcile  any  inconpatibilities  in  the  data.  For  example, 
•the  students  were  taught  -that  a  careful  examination  of  information  available 
from  -the  video  -tape  recordings  of  a  mission  could  be  used  to  identify  a  target 
engagement  -that  was  missed  by  the  real-time  casualty  assessment  system  (see 
TRADDC  Test  and  Evaluation  Comitand  Experimentation  Center,  1990) . 

Learning  station  ins-tructors  were  provided  with  -the  opportunity  to 
correct  a  variety  of  misperceptions  about  da-ta  sources.  For  instance,  many  of 
the  future  test  personnel  thought  -that  they  would  be  able  to  directly  observe 
■the  crew  while  -they  were  collecting  data.  Ihe  students  learned  from  -the 
instructors,  however,  -that  -they  would  be  viewing  video  tapes  of  the  opera-tors 
■taken  during  missions. 

Phase  Three;  Job-Specific  Training 

Phase  Ihree  of  the  -training  was  individualized,  intense,  and  job 
specific.  This  phase  is  critical  for  reliable  data  collection  because  it 
equips  -the  VDte  with  the  specific  skills  and  attitudes  needed  to  perform  -their 
jobs  conpetently.  Ihe  individualized  training  was  conducted  by  the  PVR  and 
BCA  team  leaders.  It  v.as  give.n  subsequent  to  Phases  One  and  IV'o  and  only  -to 
■those  VCRs  vho  would  be  directly  working  wi-th  -the  PVR  and  BDA  team  leaders. 
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IXiring  Phase  Ihree  of  the  training,  future  work  cx)nditions  were  siinulated 
by  using  video  tapes  from  prior  tests  of  the  IDS-F-H  system.  Prospective 
members  of  the  PVR  team  viewed  tapes  to  identify  and  document  the  actions 
taken  by  tlie  system  cperators.  Prospective  members  of  the  team  used  the 
data  recorded  by  the  PVR  team  and  their  own  viewing  of  the  video  tape  to 
synthesize  the  entire  situation,  assessing  whether  the  actions  taken  were 
correct.  If  it  was  determined  that  IDS-F-H  cperators  had  made  an  error  (i.e., 
had  not  performed  in  accordance  with  prescribed  tactics,  techniques,  and 
procedures) ,  the  BDA  video  data  reducers  were  required  to  code  the  error  and 
to  devise  a  corrective  action. 

Test  Agency  Evaluation  of  the  VDR  Training 

The  test  agency  for  the  LOS-F-H  FOTE  II  was  most  concerned  that  they  be 
able  to  have  confidence  in  the  information  they  were  given  by  the  VCRs.  As 
identified  by  Laws  and  Barber  (1989) ,  inter-rater  reliability  became  a  major 
issue  in  video  data  reduction  for  the  test  agency.  Consequently,  for  this 
IDS-F-H  test,  all  VKRs  were  required  to  perform  to  predetermined  criteria 
after  their  Phase  Three  training  and  prior  to  their  participation  in  the  FDTE. 
The  standards  were  set  by  the  testing  agency,  not  by  the  VDR  instructors  or  by 
ARI. 


The  test  agency  was  very  satisfied  with  the  high  quality  performance  of 
the  individuals  who  had  coirpleted  the  VDR  training  program.  These  students 
not  only  met  but  exceeded  the  standards  exstablished  by  the  test  agency. 
Specifically,  prospective  members  of  the  PVR  team  demonstrated  that  they  could 
correctly  reduce  a  minimum  of  five  video  tapes,  where  the  criteria  for 
correctness  were  established  prior  to  the  training.  Similarly,  future  members 
of  the  BDA  team  collectively  satisfied  previously  established  inter-rater 
reliability  standards  for  extraction  and  use  of  information  from  the  video 
tapes. 


Student  Evaluation  of  the  VDR  Training 


Method 

Students  participating  in  the  FDTE  II  data  collection  overview  (Phase 
Two)  and  the  job-specific  instruction  (Riase  Three)  were  asked  to  evaluate 
their  training.  Three  separate  questionnaires  were  developed  by  ARI  for  this 
purpose.  One  questionnaire  assessed  the  training  given  at  the  learning 
stations  during  the  Kiase  Two  data  collection  overview.  The  two  other 
questionnaires  were  administered  to  students  in  the  PVR  and  BDA  training 
^oups  (Phase  Three) ,  respectively,  at  the  conclusion  of  their  individualized 
instruction.  All  of  the  questionnaire  items  were  based  on  the  instructors' 
learning  objectives.  Copies  of  the  learning  objectives  and  questionnaires  are 
in  i^pendix  A.  Summaries  of  the  data  obtained  with  the  questionnaires  are  in 
appendix  B.  The  Ihase  One  system  overview  training  was  not  evaluated  by  ARI. 

Each  questionnaire  contained  a  series  of  items  designed  to  assess  student 
perceptions::  of  their  understanding  of  the  material  presented  during  the 
training.  Each  questionnaire  also  contained  an  item  designed  to  assess  the 
overall  value  of  the  training.  Students  responded  to  each  of  these  items 
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iKing  a  7-point  scale.  A  rating  of  ”1"  on  the  scale  indicated  that  the 
student  conrpletely  understood  the  material  presented  or  had  great  confidence 
in  his  or  her  ability  to  perform  adequately  as  a  result  of  the  training.  A 
rating  of  ”7"  on  the  scale  indicated  a  conplete  lack  of  understanding  or  a 
conplete  lack  of  confidence.  A  rating  of  "4"  iixiLcated  a  borderline  level  of 
understanding  or  confidence. 

It  ^ould  be  noted  that  the  students  vho  participated  in  the  Phase  Three 
job-specific  training  are  not  necessarily  the  same  students  v^ho  were  in  Phase 
Two.  Althou^  the  training  strategy  calls  for  the  students  to  progress  in  an 
orderly  fashion  through  the  three  phases  of  the  training,  it  was  not  possible 
to  provide  this  ej^jerience  for  all  the  VCRs  employed  during  the  LDS-F-H  FETE 
II.  Many  of  the  Phase  Three  VCR  students  had  taken  a  session  of  the  40-hour 
system  overviOT  course  prior  to  development  of  this  training  program. 
Therefore,  it  was  not  possible  for  them  to  participate  in  the  Ihase  Two 
training. 

Results  of  Student  Evaluations  of  Phcise  Two  Trzdnina 

Student  characteristics.  Eighteen  students,  11  males  and  7  females, 
participated  in  the  fhase  Two  training.  Four  of  the  females  and  four  of  the 
males  had  prior  field  test  data  collection  experience.  The  students'  mean  age 
was  37.1  years  and  their  educational  levels  ranged  from  high  school  diploma  to 
graduate  level  training  (see  Appendix  B) . 

Student  ratings.  The  mean  ratings  for  the  Phase  Two  questionnaire  items 
reflect  a  highly  positive  evaluation  of  this  phase  of  the  VDR  training;  the 
mean  ratings  ranged  from  1.8  to  2.9.  Students  indicated  that  the  training 
produced  an  understanding  of  the  issues  and  relationships  inherent  in  the 
field  test  data  collection  process. 

Comparisons  of  the  mean  ratings  as  a  function  of  the  prior  experience  and 
the  gender  of  the  students  reveal  two  meaningful  trends  in  questionnaire 
responses.  When  ratings  are  grouped  by  the  respondent's  prior  experience, 
students  who  had  previously  served  as  data  collectors  indicate  a  higher  level 
of  understanding  of  the  material  presented  than  those  with  no  such  experience. 
When  ratings  are  grouped  by  gender,  females  indicate  a  greater  understanding 
than  males  for  the  material  addressed. 

Hence,  while  all  respondents  indicate  that  the  Phase  Two  training  was 
valuable  to  their  overall  understanding  of  the  testing  environment, 
respondents  with  some  prior  experience  and  female  respondents  indicate  more 
than  the  others  that  the  training  gave  them  a  better  understanding  of  specific 
aspects  of  the  data  collection  ’'process." 

Results  of  student  Evaluations  of  Phase  Three  Trzdnina 

Student  characteristics.  Two  males  eind  five  females  participated  in  the 
training  for  PVRs,  and  three  males  participated  in  the  training  for  BDAs.  All 
Phase  Threet students  had  prior  field  data  collection  experience.  Those  who 
received  the  PVR  job-specific  training  were  generally  younger  and  had  less 
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formal  education  than  those  who  received  the  BDA  training  (see  .^:pendix  B  for 
fipther  details) , 

Student  ratings.  Ihe  mean  ratings  for  the  Phase  Three  training 
evaluation  questionnaires  show  the  students  were  very  positive  about  their  job 
specific  training.  The  students  indicated  that  they  understood  the  tasks  they 
were  expected  to  perform  and  that  they  were  confident  in  their  abilities  to 
adequately  perform  as  VESfe.  Specifically,  the  mean  response  for  the  items  to 
be  rated  in  the  PVR  training  questionnaire  ranged  from  1-2  to  2.1.  Mean 
ratings  for  the  BDA  training  questionnaire  items  ranged  from  1.0  to  2.3.  A 
brea]<down  of  these  training  evaluations  by  the  characteristics  of  the  students 
was  not  possible  due  to  the  small  sizes  of  the  two  groups  vho  received  this 
training. 


SME  Opinions  on  Personnel  Selection  Criteria 

The  selection  of  instructors  and  personnel  for  video  data  reduction  is  an 
important  consideration  in  designing  a  training  strategy.  Some  highly 
specialized  skills  are  needed  to  fill  the  role  of  instructor  but 
qualifications  can  differ  for  VDRs  according  to  the  position  to  be  filled. 

SMEs  associated  with  the  IDS-F-H  FDTE  II  constructed  a  profile  of  the 
characteristics  they  believe  necessary  for  VDR  instructors  and  students.  The 
qualities  identified  by  the  SMEs  as  being  essential  for  competent  instructors 
held  no  surprises.  There  were  some  important  differences  for  tTDRs  by 
position,  however. 

Instructor  Requirements 

The  SMEs  agreed  that  "in  the  best  of  all  possible  worlds"  video  reduction 
instructors  would  have  a  sincere  and  dedicated  desire  to  impart  the  knowledge 
to  their  students  that  will  facilitate  test  conduct.  This  quality,  of  course, 
exemplifies  professional  behavior,  SMEs  also  agree  that  instructors  imust  have 
prior  teaching  experience  and  should  be  able  to  communicate  on  a  multi-level 
basis  (i.e. ,  to  classes  comprised  of  students  with  varied  backgrounds  in  the 
military  and  in  the  test  and  evaluation  environments) .  Instructors  need  to 
became  very  familiar  with  the  capabilities  and  limitations  of  their  students 
and  utilize  this  knowledge  to  tailor  teaching  strategies.  Instructors  also 
should  have  extensive  experience  with  the  system  being  tested,  both  from  a 
maintenance  and  an  operational  perspective.  Previous  experience  in  video 
reduction  and  data  collection  is  a  must,  along  with  an  in-depth  understanding 
of  test  documentation  and  with  test  requirements  and  limitations. 

Student  Requirements 

The  SMEs  believe  that  is  it  necessary  for  BDAs  to  have  some  college 
education.  The  analyst  Tnust  have  acquired  experience  with  and  knowledge  of 
the  weapon  system  under  test  or  have,  at  a  minimum,  an  in-depth  knowledge  of 
simdlar  or  predecessor  systems.  The  analyst  should  have  a  background  in 
training,  human  factors,  or  MANPRINT.  Also  necessary  is  familiarity  with  the 
mdlitary  weapon  procureiment  si^stem.,  testing  requirements,  test  documentaticn, 
ar*d  limitations  of  the  specific  type  system  tests  (i.e.,  NDICE,  FDTE  II, 


7 


WPS,  on  the  other  hsmd,  do  not  need  the  hi^  levels  of  specialized 
skills  required  by  the  BCAs.  The  physical  requireanents  (vision,  hearing, 
etc.)  axe  the  same,  but  PVRs  need  only  a  hi^  school  educaticxi.  Othervdse,  no 
i^iecial  abilities  are  required. 

Discussion 

Althou^  an  initial  training  strategy  for  operational  field  test  VERs  hrs 
been  developed,  there  is  much  work  to  be  d(^ie  before  a  fined  strategy  is 
produced.  For  instance,  inter-rater  reliability  must  be  assessed  and  judged 
to  be  adequate  after  the  students  have  performed  their  jobs  for  some  extended 
period  of  time.  For  the  present  effort  this  assessment  was  made  prior  to 
their  work  experience  only.  The  same  is  true  of  measuring  student  perceptions 
of  satisfacticn  with  training.  Being  satisfied  with  training  prior  to 
performing  video  reduction  is  caie  matter  but  remaining  satisfied  on  the  job  is 
another.  Administration  of  student  eveduaticn  questionredres  before  actued 
training  would  provide  baseline  information  of  student  knowledge.  These  data 
could  then  be  caipared  to  ratings  given  by  the  stxxJents  at  the  conclusion  of 
trainii^  for  a  pretest-posttest  cenparison.  It  is  also  necessary  to  fonmlate 
objective  VEK  performance  standards  to  be  used  as  an  independent  gauge  of 
training  effectiveness.  Continuing  satisfaction  with  trajning  may  not 
necessarily  correlate  with  satisfactory  job  performance.  Usii^  video  tapes 
^ould  facilitate  the  development  of  performance  standarxis,  however,  because 
the  performance  of  individual  VERs  can  be  matched  to  tapes.  Team  leader 
satisfaction  with  video  reducer  performance  should  also  be  assessed. 
Instructors  and  team  leaders  need  to  be  surveyed  to  identify  arects  for  future 
improvement  in  the  video  reduction  training. 

Finally,  it  roust  also  be  demonstrated  that  the  VER  training  strategy 
enployed  actually  improves  the  validity  and  reliability  of  the  data  collected 
and  that  this  improvement  prwides  decisicai  makers  with  better  information. 
E>^icting  the  benefits  reedized  thrcui^  improved  training  will  be  the  telling 
argument  for  the  success  of  this  or  any  other  training  program. 
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APPENDIX  A 


VDR  Training  Evaluation  Questionnaires  and  VDR  Training  Objectives 

This  appendix  contains  the  student  training  evaluation 
questionnaires  and  the  training  objectives  for  each  of  the  three 
types  of  video  data  reduction  training.  The  following  will  assist 
in  the  use  of  this  appendix: 

Questionnaire  for 

Phase  Two  Overview  Training  . . A-2  through  A-6 

Learning  Objectives  for 

Phase  Two  Overview  Training  .  A-7  and  A-8 

Questionnaire  for  Phase  Three 

Performance  Video  Reducer  (PVR)  Training  . . .  A-9  through  A-14 


Learning  Objectives  for 

Phase  Three  PVR  Training  .  A- 15 

Questionnaire  for  Phase  Three 

Battle  Drill  Analyst  (BDA)  Training  .  A-16  through  A-24 

Learning  Objectives  for 

Phase  Three  BDA  Training .  A-25  and  A-26 
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LOS-P-H  DATA  COLLECTOR  AND  VIDEO  REDUCER 
TRAINING  ASSESSMENT  QUESTIONNAIRE 


FDTE  II  has  the  role  of  testing  tactics,  techniques  and  procedures 
up  to  and  including  the  platoon  level.  The  training  you  received 
at  the  learning  stations  had  as  one  of  its  purposes  fostering  an 
appreciation  within  the  testing  community  for  the  interrelationship 
ot  the  Toles  played  by  the  various  testing  organizations  during 
FDTE  II.  Other  goals  of  the  learning  station  instruction  were  to 
give  you  an  understanding  of  performance,  RAM,  and  battle  drill 
analysis  data  available  through  the  video  reduction  process,  and 
to  familiarize  you  with  TEC  data  collection  and  video  reduction 
instruments.  We  hope  that  each  of  you,  experts  within  your  own 
areas,  now  have  a  better  idea  of  how  your  job  affects  the  testing 
process,  relates  to,  and  affects  other  areas. 

The  purpose  of  this  questionnaire  is  to  assess  the  effectiveness 
of  the  training  you  received  at  the  learning  stations.  Your 
responses  to  this  questionnaire  will  be  used  to  evaluate  the 
learning  station  instruction  for  the  purpose  of  improving  and 
upgrading  it  for  future  courses . 

We  have  enjoyed  working  with  you  and  we  appreciate  the  time  and 
effort  you  have  put  into  the  course  and  into  answering  this 
questionnaire . 


A.  Sex:  M  F  Age  _ 

B.  What  is  your  expected  job  on  the  upcoming  test? 

RAM  data  collector  _ 

Performance  video  reducer  _ 

Battle  drill  analyst  _ 

Analyst  _ 

Other  (please  specify)  _ 

C.  Have  you  ever  worked  as  a  video  reducer,  data  collector,  or 
analyst  before? 

Yes  No 


D.  If  yes,  did  you  work  as  a 

Data  collector  _ 

Video  reducer  _ 

Analyst  _ 
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E.  If  yes,  what  was  the  name  of  the  system (s)  or  other  equipment 
on  which  you  worked  as  a  data  collector,  video  reducer,  or  analyst? 


F.  What  is  the  highest  level  of  education  you  have  attained? 

8  yrs.  _  9  yrs.  _  10  yrs.  _ 

11  yrs.  _  GED  _  High  school  diploma  _ 

1  yr.  college  _  2  yrs.  college  _ 

3  yrs. college  _  College  degree  _ 

Graduate  training:  Yes  No 


Please  use  the  following  code  to  respond  to  questions  1  through 

11. 


1.  Very  completely 

2.  Quite  completely 

3 .  Fairly  completely 

4.  Borderline 

5.  Fairly  incompletely 

6 .  Quite  incompletely 

7 .  Very  incompletely 

1.  I  understand  the  symbology  on  the  thru-sight  and  PPI . 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


2.  I  understand  the  engagement  sequence  file  record  format. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 
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I  understand  the  sources  of  data  for  data  elements 


COMMENTS : 


4.  I  understand  the  relationships  between  sources  of  data. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _ 


5.  I  understand  the  sequence  of  events  which  take  place  during 
engagement . 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS;  _ 


6.  I  understand  the  relevance  of  test  data  as  they  apply  to  live 
fire. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


7.  I  understand  the  relationship  between  the  MANPRINT  video 
reduction  forms  and  the  engagement  data. 

1  2  3  4  5  6  7 


COMMENTS 


8.  I  understand  the  relationship  between  MANPRINT  video  reduction 
forms  and  RAM  data. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _ 


9.  I  understand  the  relationship  between  MANPRINT  video  reduction 
forms  and  tactics  results. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS :  _  _  _ 


10.  I  understand  the  relationship  between  MANPRINT  and  test 
conduct . 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _ _ _ 


11.  I  understand  the  system  through  which  fixes  are  made  to 
tactics,  techniques  and  procedures  using  engagement  and  MANPRINT 
data . 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 
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12.  I  can  associate  the  actions  of  one  battle  drill  with  its 
impact  on  another  battle  drill. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS :  _ _ _  _  _ 


13.  How  valuable  was  the  training  you  received  at  the  learning 
stations  in  increasing  your  understanding  of  the  interrelated  roles 
of  the  testing  community? 

1 _ 2__ _ 3 _ 4 _ 5 _  6  _ 7 

Of  No 
Value 

COMMENTS : 


Very 

Valuable 


14 .  Please  add  any  comments  which  will  help  us  in  improving  and 
upgrading  this  course.  Thank  you. 


LEARKIN6  STATION  #1  OBJECTIVES 


1.  To  understand  the  symbology  on  the  thru-sight  and  PPI. 

2.  To  understand  the  engagement  sequence  file  record  format. 

3.  To  understand  the  sources  of  data  for  data  elements. 

4.  To  understand  the  relationships  between  sources  of  data. 

5.  To  understand  the  sequence  of  events  which  take  place  during 
engagement . 

6.  To  understand  the  relevance  of  test  data  as  they  apply  to  live 
fire. 


LEARNING  STATION  #2  OBJECTIVES 


1 .  To  understand 
reduction  forms  and 

the 

the 

relationship  between 
engagement  data. 

the 

MANPRINT 

video 

2,  To  understand 

reduction  forms  and 

the 

RAM 

relationship 

data. 

between 

the 

MANPRINT 

video 

3 .  To  understand 
reduction  forms  and 

the  relationship 
tactics  results. 

between 

the 

MANPRINT 

video 

4 .  To  understand 

conduct. 

the 

relationship 

between 

MANPRINT  and 

test 

5.  To  understand  the  system  through  which  fixes  are  made  to 
tactics,  techniques  and  procedures  using  engagement  and  MANPRINT 
data. 

6.  To  associate  the  actions  of  one  battle  drill  with  its  impact 
on  another  battle  drill. 
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LOS-F-H  PERFORMANCE  VIDEO  REDUCER 
TRAINING  ASSESSMENT  QUESTIONNAIRE 


The  purpose  of  this  questionnaire  is  to  assess  the  effectiveness 
of  the  performance  video  reducer  training.  This  training 
represents  an  effort  to  upgrade  and  improve  the  training  received 
by  performance  video  reducers  in  previous  test  situations.  Your 
responses  to  this  •questionnaire  will  enable  us  to  evaluate  the 
performance  video  reducer  training  procedures  and  to  make  further 
improvements  to  the  training  where  necessary.  Thank  you  for  your 
cooperation. 

DEMOGRAPHIC  INFORMATION 


A. 

Sex: 

M 

F 

B. 

Age 

C. 

Have 

you 

ever  worked  as  a  video  reducer,  data  collector  or 

analyst  before? 


Yes  No 


D.  If  yes  to  C,  did  you  work  as  a 

Data  collector  _ 

Video  reducer  _ 

Analyst  _ 

E.  If  yes  to  C,  what  was  the  name  of  the  system (s)  or  other 
equipment  on  which  you  worked? 


F.  What  is  the  highest  level  of  education  you  have  attained? 

8  yrs.  _  9  yrs.  _  10  yrs.  _ 

11  yrs.  _  GED  _  High  school  diploma  _ 

1  yr.  college  _  2  yrs.  college  _ 

3  yrs.  college  _  College  degree  _ 

What  is  your  degree  discipline?  _ 

Graduate  training:  Yes  No 
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Please  use  this  code  to  respond  to  the  questions  which  follow. 


1.  Very  completely 

2.  Quite  completely 

3.  Fairly  completely 

4.  Borderline 

5.  Fairly  incompletely 

6 .  Quite  incompletely 

7 .  Very  incompletely 

1.  I  can  differentiate  among  the  various  target  types  (friendly, 
hostile,  ground  targets) . 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS: _ _  _  _ _ 


2.  I  can  record  the  time  of  initial  target  detect. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


3.  I  can  identify  the  target  detection  system  (FLIR,  TV,  Radar, 
visual) . 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


4.  I  can  identify  the  detector  of  the  target  (EO,  RO,  Driver). 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 
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5.  I  can  identify  the  source  of  target  information  (RODS,  Netted, 
MSCS,  autonomous) . 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS;  _ 


6.  I  know  what  data  to  use  to  determine  if  an  aircraft  is 
ingressing  or  egressing. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS :  _  _ _ 


7.  I  know  the  3  codes  which  apply  when  trying  to  identify  whether 
or  not  the  target  was  on  the  PPI  at  initial  detection. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS ; 


8.  I  can  identify  track  designations  of  friend,  foe,  or  unknown 
on  the  radar. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


9.  I  can  differentiate  between  raw  and  symbolic  video. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS :  ■  _ 


A-11 


10.  I  can  identify  the  time  when  handoff  is  initiated. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _ 


11.  I  can  identify  the  time  of  EO  handoff  completion. 

1 _ 7. _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS :  _ _ 


12.  I  can  identify  the  time  of  RO  handoff  completion. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS :  _ 


13.  I  can  identify  the  time  of  EO  acquire. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


14.  I  can  identify  the  time  of  EO  track  initiate. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 
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15.  I  am  aware  of  aircraft  tactics  (jinking,  direct  approach, 
hovering,  running,  unknown) . 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : _ _ 


16.  I  can  record  the  time  of  target  ID. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


17.  I  can  record  the  time  of  the  engagement  termination. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS ; 


18.  I  can  record  the  reason  icr  “orl/  engagement  termination  using 
situation  codes. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


19.  I  can  record  the  last  event  completed  before  engagement 
termination. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 
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20.  I  know  the  situation  codes  for  each  data  element. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _ _ _ 


21.  Do  you  feel  that  sufficient  time  was  allowed  for  your  training 
as  a  performance  video  reducer  prior  to  FDTE  II? 

Yes  No  _ 


COMMENTS : 


22.  What  training,  if  any,  have  you  received  in  the  past  that  made 
this  learning  experience  easier  for  you? 


23.  Please  rate  your  confidence  in  your  ability  to  perform  your 
duties  as  a  performance  video  reducer  based  the  training  you 
received. 

1 _ 2 _ ^3 _ 4 _ 5 _ 6 _ 7 

Total  Lack  of 
Confidence 

COMMENTS :  _ _ 


Very 

Confident 


24.  Please  add  any  additional  comments  which  will  assist  us  in 
improving  and  upgrading  this  training. 
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PERFORMANCE  VIDEO  REDUCER  LEARNING  OBJECTIVES 


1.  To  be  able  to  differentiate  among  the  various  target  types 
(friendly,  hostile,  ground  targets). 

2.  To  be  able  to  record  the  time  of  initial  target  detect. 

3.  To  Idb  •ablo  to  identify  the  target  detection  system  (FLIR,  TV, 
Radar,  visual) . 

4.  To  be  able  to  identify  the  detector  of  the  target  (EO,  RO, 
Driver)  . 

5.  To  be  able  to  identify  the  source  of  target  information  (RODS, 
Netted,  MSCS,  Autonomous). 

6.  To  know  the  data  used  to  determine  if  an  aircraft  is  ingressing 
or  egressing. 

7.  To  know  the  3  codes  which  apply  when  trying  to  identify  whether 
or  not  the  target  was  on  the  PPI  at  initial  detection. 

8.  To  be  able  to  identify  the  track  designations  of  friend,  foe, 
or  unknown  on  the  radar. 

9.  To  be  able  to  differentiate  between  raw  and  symbolic  video. 

10.  To  be  able  to  identify  the  time  when  handoff  is  initiated. 

11.  To  be  able  to  identify  the  time  of  EO  handoff  completion. 

12.  To  be  able  to  identify  the  time  of  RO  handoff  completion. 

13.  To  be  able  to  identify  the  time  of  EO  acquire. 

14.  To  be  able  to  identify  the  time  of  EO  track  initiate. 

15.  To  be  aware  of  aircraft  tactics  (jinking,  direct  approach, 
hovering,  running,  unknown) . 

16.  To  be  able  to  record  the  time  of  target  ID. 

17.  To  be  able  to  record  the  time  of  engagement  termination. 

18.  To  be  able  to  record  the  reason  for  early  engagement 
termination,  using  situation  codes. 

19.  To  be  able  to  record  the  last  event  completed  before  engagement 
termination. 

20.  To  know  the  situation  codes  for  each  data  element. 
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LOS-F-H  BATTLE  DRILL  ANALYST 
TRAINING  ASSESSMENT  QUESTIONNAIRE 


The  purpose  of  this  questionnaire  is  to  assess  the  effectiveness 
of  the  Battle  Drill  Analyst  training.  This  training  represents  an 
effort  to  upgrade  and  improve  the  training  received  by  data 
collectors  in  previous  field  test  situations.  Your  responses  to 
this  questionnaire  will  enable  us  to  evaluate  the  Battle  Drill 
Analyst  training  procedures  and  to  make  further  improvements  to  the 
training  where  necessary.  Thank  you  for  your  cooperation. 

DEMOGRAPHIC  INFORMATION 

A.  Sex:  M  F 

B .  Age  _ 

C.  Have  you  ever  worked  as  a  video  reducer,  data  collector,  or 
analyst  before? 

Yes  _  No  _ 


D.  If  yes  to  C,  did  you  work  as  a 

Data  collector  _ 

Video  reducer  _ 

Analyst  _ 

E.  If  yes  to  C,  what  was  the  name  of  the  system (s)  or  other 
equipment  on  which  you  worked? 


F.  What  is  the  highest  level  of  education  you  have  attained? 

8  yrs.  _  9  yrs.  _  10  yrs.  _ 

11  yrs.  _  GED  _  High  school  diploma  _ 

1  yr.  college  _  2  yrs.  college  _ 

3  yrs.  college  _  College  degree  _ 

What  is  your  degree  discipline?  _ 

Graduate  training:  Yes  _  No  _ 
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Please  use  this  code  to  respond  to  the  questions  which  follow. 


1.  Very  completely 

2 .  Quite  completely 

3 .  Fairly  completely 

4.  Borderline 

5.  Fairly  incompletely 

6.  Quite  incompletely 

7.  Very  incompl-etely 

1.  I  am  familiar  with  all  the  documents  being  tested  (eg. ,  STP, 
FMs,  PDEPs) . 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS :  _ _ _ 


2.  I  am  expert  in  applying  the  6  standardized  MANPRINT  codes. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _ 


3.  I  am  expert  in  the  9  crew  performance  categories. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


4.  I  am  a  LOS-F-H  system  SME  with  regard  to  maintainer-operator 
interface. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS ;  _ _ _ 
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5.  I  am  familiar  with  the  test-fix-test  concept  and  the  process 
of  evaluating  recommended  fixes. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


6.  I  am  expert  in  engagement  video  reduction. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


7.  I  am  aware  of  the  principles  for  tactical  deployment  of  the 
LOS-F-H  system. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS;  _ 


8.  I  am  aware  of  the  organizational  structure  for  the  LOS-F-H. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS ; 


9.  I  am  able  to  recognize  and  research  the  operational  effects 
of  operator  error. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 
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10.  I  know  how  to  trace  the  source  of  the  operator's  required 
actions  during  tactical  deployment,  which  includes  performance  of 
most  battle  drills. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _ _ _ _ _ 


11.  I  am  expert  in  the  procedures  required  for  Prepare  for  March 
Order. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _ 


12.  I  am  expert  in  the  procedures  required  for  Emplace  and  Prepare 
for  Action. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS :  _ _ _ 


13.  I  am  expert  in  the  procedures  required  for  Perform  Engagement 
Against  Hostile  Aircraft, 

1  2  3  4  _  5 _  6 _ 7 


COMMENTS : 


14 .  I  am  expert  in  the  procedures  required  for  Perform  Hasty 
Engagement  Against  Hostile  Aircraft. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 1 

COMMENTS :  _ _ 


15-  I  am  expert  in  the  procedures  required  for  Perform  Netted 
Engagement  Against  Hostile  Aircraft. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : _ 


16.  I  am  expert  in  the  procedures  required  for  Perform  Degraded 
Mode  Against  Hostile  Aircraft. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS :  _  _ 


17.  I  am  expert  in  the  procedures  required  to  perform  Hangfire  or 
Misfire  procedure. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 
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18.  I  am  able  to  recognize  and  evaluate  the  abbreviated  Rearm 
procedures . 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS;  _ 


19.  I  can  define  and  explain  the  crew  performance  error  code 
'•Procedure  omitted”. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS: _ _ 


20.  I  can  define  and  explain  the  crew  performance  error  code 
"Procedure  performed  inaccurately". 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS :  _ _ 


21.  I  can  define  and  explain  the  crew  performance  error  code 
"Procedure  performed  at  the  wrong  time". 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 
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22.  I  can  define  and  explain  the  crew  performance  error  code 
"Wrong  procedure  performed”. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _  _ _ 


23.  I  can  define  and  explain  the  crew  performance  error  code 
"Extra  procedure  performed" . 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS: _ _ _ 


24.  I  can  define  and  explain  the  crew  performance  error  code 
"Procedure  delay". 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _ 


25.  I  am  able  to  use  all  the  data  provided  by  multiple  sources  to 
determine  the  focus  for  recognizing  operator  interface  errors. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _ _ 
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26.  I  am  familiar  with  applying  OTEA  priority  codes  to  crew  error. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS:  _ 


27.  I  am  able  to  suggest  or  recommend  corrections  to  existing 
tactics,  techniques  and  procedures. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


28.  Do  you  feel  sufficient  time  was  allowed  for  your  training  as 
a  Battle  Drill  Analyst  prior  to  FDTE  II? 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

COMMENTS : 


29.  What  training,  if  any,  have  you  received  in  the  past  that  made 
this  learning  experience  easier  for  you? 


30.  Based  on  your  understanding  of  what  is  to  be  tested  at  the 
Fire  Unit  level,  how  many  people  would  you  estimate  are  needed  to 
perform  your  functions  for  each  Fire  Unit? 


31.  Please  rate  your  confidence  in  your  ability  to  perform  your 
duties  as  a  Battle  Drill  Analyst  based  on  the  training  you  have 
received. 

1 _ 2 _ 3 _ 4 _ 5 _ 6 _ 7 

Total  Lack  of 
Confidence 

COMMENTS :  _ 


Very 

Confident 


32.  Please  add  any  additional  comments  which  will  assist  us  in 
improving  and  upgrading  this  training. 


BATTLE  DRILL  ANALYST  LEARNING  OBJECTIVES 


1.  TO  become  familiar  with  all  the  documents  being  tested  (e.g., 
STP,  FMs,  PDEPs) . 

2.  To  become  expert  in  applying  the  6  standardized  MANPRINT 
codes. 

3.  To  become  expert  in  the  9  crew  performance  categories. 

4.  To  become  a  LOS-F-H  system  SME  with  regard  to  maintainer- 
operator  interface. 

5.  To  become  familiar  with  the  test-fix-test  concept  and  the 
process  of  evaluating  recommended  fixes. 

6.  To  become  expert  in  engagement  video  reduction. 

7.  To  become  aware  of  the  principles  for  tactical  deployment  of 
the  LOS-F-H  system. 

8.  To  become  aware  of  the  organizational  structure  for  the  LOS- 
F-H  platoon. 

9.  To  be  able  to  recognize  and  research  the  operational  effects 
of  operator  error. 

10.  To  know  how  to  trace  the  source  of  the  operator's  required 
actions  during  tactical  deployment,  which  includes  performance  of 
most  battle  drills. 

11.  To  become  expert  in  the  procedures  required  for  Prepare  for 
March  Order. 

12.  To  become  expert  in  the  procedures  required  for  Emplace  and 
Prepare  for  Action. 

13.  To  become  expert  in  the  procedures  required  for  Perform 

Engagement  Against  Hostile  Aircraft. 

14.  To  become  expert  in  the  procedures  required  for  Perform  Hasty 
Engagement  Against  Hostile  Aircraft. 

15.  To  become  expert  in  the  procedures  required  for  Perform  Netted 
Engagement  Against  Hostile  Aircraft. 

16.  To  become  expert  in  the  procedures  required  for  Perform 

Degraded  Mode  Engagement  Against  Hostile  Aircraft. 

17.  To  become  expert  in  the  procedures  required  to  perform 

Hangfire  or  Misfire  procedure. 
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18.  To  be  able  to  recognize  and  evaluate  the  abbreviated  Rearm 
procedures. 

19.  To  be  able  to  define  and  explain  the  crew  performance  error 
code  ’’Procedure  omitted". 

20.  To  be  able  to  define  and  explain  the  crew  performance  error 
code  "Procedure  performed  inaccurately". 

21.  To  be  able  to  define  and  explain  the  crew  performance  error 
code  "Procedure  performed  at  the  wrong  time". 

22.  To  be  able  to  define  and  explain  the  crew  performance  error 
code  "Wrong  procedure  performed". 

23.  To  be  able  to  define  and  explain  the  crew  performance  error 
code  "Extra  procedure  performed". 

24.  To  be  able  to  define  and  explain  the  crew  performance  error 
code  "Procedure  delay". 

25.  To  be  able  to  use  all  the  data  provided  by  multiple  sources 
to  determine  the  focus  for  recognizing  operator  interface  errors. 

26.  To  become  familiar  with  applying  OTEA  priority  codes  to  crew 
error. 

27.  To  be  able  to  suggest  or  recommend  corrections  to  existing 
tactics,  techniques  and  procedures. 


APPENDIX  B 


summaries  of  student  Evaluations  of  Phase  Two  Overview  Training, 
Phase  Three  Performance  Video  Reducer  (PVR) 

And  Battle  Drill  Analyst  (BDA)  Training 

This  appendix  contains  summaries  of  the  student  evaluations 
of  Phase  Two  and  Phase  Three  training.  The  following  will  assist 


in  the  use  of  this  appendix: 

Summary  of  Student  Evaluations  of 

Phase  Two  Overview  Training  .  B-2  through  B-6 

Summary  of  Student  Evaluations  of  Phase  Three 

Performance  Video  Reducer  Training  .  B-7  through  B-10 

Summary  of  Student  Evaluations  of  Phase  Three 

Battle  Drill  Analyst  Training  .  B-11  through  B-15 
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SUMMARY  OF  STUDENT  EVALUATIONS  OF 
PHASE  TWO  OVERVIEW  TRAINING 


Eighteen  students  responded  to  the  questionnaire — ll  males  and  7 
females.  The  mean  age  is  37.13  years,  and  ages  range  from  19  to 
60  years.  The  students  filled  a  variety  of  data  collection  roles 
during  FDTE  II,  including  reliability,  availability,  and 
maintainability  (SAM) ,  performance  video  reducer,  and  battle 
drill  analyst  data  collectors.  Four  males  and  four  females  had 
prior  field  test  data  collection  experience.  Six  students  hold 
high  school  diplomas  or  the  equivalent,  six  have  attended 
college,  three  hold  college  degrees,  and  three  have  graduate 
training. 

The  items  which  comprise  the  phase  two  questionnaire  appear 
below.  They  are  accompanied  by  the  means  and  standard  deviations 
for  each  item,  first  by  gender  and  then  by  experience.  Mean 
ratings  by  gender  or  by  experience  that  are  significantly 
different  are  shown  by  an  asterisk  In  both  types  of 
comparisons,  the  test  statistic  was  the  t-test. 

Means  and  standard  Deviations.  Combined  and  bv  Sex 

1.  I  understand  the  symbology  on  the  thru-sight  and  PPI . 


Mean 

SD 


COMBINED  MALES 

2.94  3.00 

.78  .95 


FEMALES 

2.83 

.37 


2.  I  understand  the  engagement  sequence  file  record  format. 


COMBINED  MALES 
Mean  2.83  2.91 
SD  1.01  1,08 


FEMALES 

2.67 

.94 


3  . 


I  understand  the  sources  of  data 


for  data  elements. 


COMBINED  MALES 
Mean  2.83  3.00 
SD  .83  .74 


FEMALES 

2.50 

.96 


4. 


I  understand  the  relationships  between  sources  of  data. 


Mean 

SD 


COMBINED 

2.83 

.96 


MALES 

3.09 

.90 


FEMALES 
2 . 50 
.  96 
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5.  I  understand  the  sequence  of  events  which  take  place  during 
engagement. 


Mean 

SD 

COMBINED 

2.50 

.83 

MALES 

2.45 

.89 

FEMALES 

2.83 

.37 

6. 

I  understand  the  relevance  of 
live  fire. 

test  data  as 

they  apply  to 

Mean 

SD 

COMBINED 

2.22 

1.03 

MALES 

2.36 

.98 

FEMALES 

2.17 

1.07 

7. 

I  understand  the  relationship  between  the 
reduction  forms  and  the  engagement  data. 

MANPRINT 

video 

Mean 

SD 

COMBINED 

2.78 

1.08 

MALES 

3.27 

.96 

FEMALES 

2.00 

.82 

* 

8. 

I  understand  the  relationship 
reduction  forms  and  RAM  data. 

between  MANPRINT 

video 

Mean 

SD 

COMBINED 

2.61 

1.11 

MALES 

2.91 

1.08 

FEMALES 

2.17 

1.07 

9. 

I  understand  the  relationship  between  MANPRINT 
reduction  forms  and  tactics  results. 

video 

Mean 

SD 

COMBINED 
2.56 
.  90 

MALES 

2.91 

.79 

FEMALES 

2.00 

.82 

* 

10. 

I  understand  the  relationship 
conduct. 

between  MANPRINT 

and 

test 

Mean 

SD 

COMBINED 

2.50 

.90 

MALES 

2.80 

.83 

FEMALES 
2.00 
.  82 

* 

11.  I  understand  the  system  through  which  fixes  are  made  to 
tactics,  techniques  and  procedures  using  engagement  and 
MANPRINT  data. 

COMBINED 

Mean  2.72 

SD  '  .99 


MALES  FEMALES 

2.73  2.83 

1.05  .90 
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12.  I  can  associate  the  actions  of  one  battle  drill  with  its 
impact  on  another  battle  drill. 

COMBINED  MALES  FEMALES 

Mean  2.33  2.09  2.83 

SD  .94  .79  1.07 

13.  How  valuable  was  the  training  you  received  at  the  learning 
stations  in  increasing  your  understanding  of  the 
interrelated  roles  of  the  testing  community? 

COMBINED  MALES  FEMALES 

Mean  1.94  2.00  1.83 

SD  1.08  1.21  .90 


Means  and  Standard  Deviations.  Combined  and  by  Experience 


1.  I  understand  the  symbology  on  the  thru-sight  and  PPI. 


Mean 

SD 


COMBINED 

2.94 

.78 


EXPERIENCE 

2.75 

.43 


NO  EXPERIENCE 
3.20 
.87 


I  understand  the  engagement  sequence  file  record  format. 


Mean 

SD 


COMBINED 

2.83 

1.01 


EXPERIENCE 

2.25 

.83 


NO  EXPERIENCE 
3.30 
.90 


I  understand  the  sources  of  data  for  data  elements. 


Mean 

SD 


COMBINED 

2.83 

.83 


EXPERIENCE 

2.38 

.86 


NO  EXPERIENCE 
3.20 
.  60 


I  understand  the  relationships  between  sources  of  data. 


Mean 

SD 


COMBINED 

2.83 

.96 


EXPERIENCE 

2.50 

.87 


NO  EXPERIENCE 
3.10 
.94 


I  understand  the  sequence  of  events  which  take  place  during 
engagement . 


Mean 

SD 


COMBINED 

2.50 

.83 


EXPERIENCE 

2.50 

.71 


NO  EXPERIENCE 
2.70 
.78 
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6.  I  understand  the  relevance  of  test  data  as  they  apply  to 
live  fire. 


Mean 

SD 


COMBINED 

2.22 

1.03 


EXPERIENCE 

1.75 

.97 


NO  EXPERIENCE 
2.60 
.92 


I  understand  the  relationship  between  the  MANPRINT  video 
reduction  forms  and  the  engagement  data. 


Mean 

SD 


COMBINED 

2.78 

1.08 


EXPERIENCE 

2.13 

.78 


NO  EXPERIENCE 
3.30 
1.00 


I  understand  the  relationship  between  MANPRINT  video 
reduction  forms  and  RAM  data. 


Mean 

SD 


COMBINED 

2.61 

1.11 


EXPERIENCE 

2.13 

1.05 


NO  EXPERIENCE 
3.00 
1.00 


I  understand  the  relationship  between  MANPRINT  video 
reduction  forms  and  tactics  results. 


Mean 

SD 


COMBINED 

2.56 

.90 


EXPERIENCE 

2.13 

.78 


NO  EXPERIENCE 
2.90 
.83 


I  understand  the  relationship  between  MANPRINT  and  test 
conduct. 


Mean 

SD 


COMBINED 

2.50 

.90 


EXPERIENCE 

2.13 

.93 


NO  EXPERIENCE 
3 . 00 
.  63 


I  understand  the  system  through  which  fixes  are  made  to 
tactics,  techniques,  and  procedures  using  engagement  and 
MANPRINT  data. 


Mean 

SD 


COMBINED 

2.72 

.99 


EXPERIENCE 

2.25 

.66 


NO  EXPERIENCE 
3 . 10 
1.04 


I  can  associate  the  actions  of  one  battle  drill  with  its 
impact  on  another  battle  drill. 


Mean 

SD 


COMBINED 

2.33 

.94 


EXPERIENCE 

1.88 

.78 


NO  EXPERIENCE 
2.60 
.92 
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13.  HOW  valuable  was  the  training  you  received  at  the  learning 
stations  in  increasing  your  understanding  of  the 
interrelated  roles  of  the  testing  community? 


Mean 

SD 


COMBINED 

1.94 

1.08 


experience 

1.38 

.70 


NO  EXPERIENCE 
2.40 
1.11 


SUMMAKY  OF  STUDENT  EVALUATIONS  OF  PHASE  THREE 
PERFORMANCE  VIDEO  REDUCER  TRAINING 


Eight  students  responded  to  the  questionnaire — 2  males  and  6 
females.  Ages  ranged  from  19  to  31  years,  with  the  mean  age 
being  25  years.  All  students  had  prior  field  test  data 
collection  experience.  Two  respondents  hold  high  school  diplomas 
and  1  has  a  college  degree.  The  other  5  have  attended  college 
for  various  periods,  but  do  not  hold  degrees. 

Questions.  Means  and  Standard  Deviations 

1.  I  can  differentiate  among  the  various  target  types 
(friendly,  hostile,  ground  targets) . 

Mean  1.74 
SD  ,83 

2.  I  can  record  the  time  of  initial  target  detect. 

Mean  1.38 
SD  .48 

3.  I  can  identify  the  target  detection  system  (FLIR,  TV,  Radar, 
visual) . 

Mean  1.63 
SD  .86 

4.  I  can  identify  the  detector  of  the  target  (EO,  RO,  Driver). 

Mean  1.63 
SD  .70 

5.  I  can  identify  the  source  of  target  information  (RDDS, 
Netted,  MSCS,  autonomous). 

Mean  1.88 
SD  .78 

6.  I  know  what  data  to  use  to  determine  if  an  aircraft  is 
ingressing  or  egressing. 


Mean  2.00 
SD  1.00 
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7.  I  know  the  3  codes  which  apply  when  trying  to  identify 
whether  or  not  the  target  was  on  the  PPI  at  initial 
detection. 

Mean  1.63 
SD  .70 

8.  I  can  identify  track  designations  of  friend,  foe,  or  unknown 
on  the  radar. 

Mean  1.50 
SD  .50 

9.  I  can  differentiate  between  raw  and  symbolic  video. 

Mean  1.63 
SD  .86 

10.  I  can  identify  the  time  when  handoff  is  initiated. 

Mean  1.25 
SD  .43 

11.  I  can  identify  the  time  of  EO  handoff  completion. 

Mean  1.25 
SD  .43 

12.  I  can  identify  the  time  of  RO  handoff  completion. 

Mean  1.25 
SD  .43 

13.  I  can  identify  the  time  of  EO  acquire. 

Mean  1.25 
SD  .43 

14.  I  can  identify  the  time  of  EO  track  initiate. 

Mean  1.25 
SD  .43 

15.  I  am  aware  of  aircraft  tactics  (jinking,  direct  approach, 
hovering,  running,  unknown) . 

Mean  1.75 
SD  .83 
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16.  I  can  record  the  time  of  target  ID. 

Mean  1.25 
SD  .43 

17.  I  can  record  the  time  of  the  engagement  termination. 

Mean  1.50 
SD  .71 

18.  I  can  record  the  reason  for  early  engagement  termination 
using  situation  codes. 

Mean  1.63 
Sd  .86 

19.  I  can  record  the  last  event  completed  before  engagement 
termination. 

Mean  1.38 
SD  .48 

20.  I  know  the  situation  codes  for  each  data  element. 

Mean  2.13 
SD  ,93 

Items  21  through  24  were  included  to  evaluate  other  aspects  of 
the  training.  A  summary  of  these  items,  including  student 
comments,  follows. 

21.  Do  you  feel  that  sufficient  time  was  allowed  for  your 
training  as  a  performance  video  reducer  prior  to  FDTE  II? 

Yes:  8  No:  0 

Student  Comments; 

During  the  one  week  personal  training  we  were  able 

to  ask  questions  and  go  through  the  process  of  operating  the 

vehicle  using  layouts. 

Fort  Bliss  training  alone  would  not  have  been  enough.  The 
intensive  personal  training  conducted  here  at  Fort 
Hunter-Liggett  has  been  most  productive. 

22.  What  training,  if  any,  have  you  received  in  the  past  that 
made  this  learning  experience  easier  for  you? 

Video  reducer  on  PMS  FDTE  II 

Video  reducer  on  LOS-F-H  FDTE  I;  training  on  operating  the 
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LOS-F-H 


PMS 

Reduction  of  video  on  FDTE  II 

PMS 

PMS 

PMS 

23.  Please  rate  your  confidence  in  your  ability  to  perform  your 
duties  as  a  performance  video  reducer  based  on  the  training 
you  have  received. 

Mean  1.38 
SD  .48 

Student  Comments; 

I  found  the  instructor  to  be  very  confident  in  the  knowledge 
he  passed  on  to  us  in  the  training  of  the  system.  He  was 
also  very  easy  to  understand  and  follow  in  the  training 
sessions. 

I  feel  that  this  training  will  be  extremely  helpful. 

Our  instructor  is  very  skilled.  This  in  turn  gave  us  some 
good  “heads  up"  training.  The  classes  were  extremely 
helpful  to  me. 

The  fact  that  our  instructor  will  be  here  to  answer  our 
questions  during  video  reduction  is  also  a  big  help. 

24.  Please  add  any  additional  comments  which  will  assist  us  in 
improving  and  upgrading  this  training. 

There  is  no  need  to  upgrade  this  training. .. .however ,  if 
our  instructor  is  on  the  test  of  a  new  system,  send  him,  he 
is  very  good. 

Our  instructor  did  a  great  job! 
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SUMMARY  OF  STUDENT  EVALUATIONS  OF  PHASE  THREE 
BATTLE  DRILL  ANALYST  (BDA)  TRAINING 


Three  males,  ages  28,  42  and  45  responded  to  the  questionnaire. 
All  had  prior  experience  as  field  test  data  collectors.  Two  hold 
college  degrees  and  one  has  had  three  years  of  college. 

Questions,  Means  and  Standard  Deviations 

1.  I  am  familiar  with  all  the  documents  being  tested  (e.g. , 

STP,  FMs,  PDEPs) . 

Mean  2.00 
SD  .82 

2.  I  am  expert  in  applying  the  6  standardized  MANPRINT  codes. 

Mean  1.67 
SD  .47 

3.  I  am  expert  in  the  9  crew  performance  categories. 

Mean  1.67 
SD  .94 


I  am  a  LOS-F-H  system  SME  with  regard  to  maintainer-operator 
interface. 


Mean  2.67 
SD  .47 

5.  I  am  familiar  with  the  test-fix- test  concept  and  the  process 
of  evaluating  recommended  fixes. 

Mean  1.33 
SD  .47 

6.  I  am  expert  in  engagement  video  reduction. 

Mean  2.00 
SD  .82 

7.  I  am  aware  of  the  principles  for  tactical  deployment  of  the 
LOS-F-H  system. 

Mean  1.67 
SD  .94 
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8.  I  am  aware  of  the  organizational  structure  for  the  LOS-F-H 
platoon. 

Mean  1.33 
SD  .47 

9.  I  am  able  to  recognize  and  research  the  operational  effects 
of  operator  error. 

Mean  1.67 
SD  .47 

10.  I  know  how  to  trace  the  source  of  the  operator's  required 
actions  during  tactical  deployment  which  includes 
performance  of  most  battle  drills. 

Mean  1.33 
SD  .47 

11.  I  am  expert  in  the  procedures  required  for  March  Order. 

Mean  1.00 
SD  .00 

12 .  I  am  expert  in  the  procedures  required  for  Emplace  and 
Prepare  for  Action. 

Mean  1.67 
SD  .94 

13.  I  am  expert  in  the  procedures  required  for  Perform 
Engagement  Against  Hostile  Aircraft. 

Mean  1.00 
SD  .00 

14 .  I  am  expert  in  the  procedures  required  for  Perform  Hasty 
Engagement  Against  Hostile  Aircraft. 

Mean  1.67 
SD  .47 

15.  I  am  expert  in  the  procedures  required  for  Perform  Netted 
Engagement  Against  Hostile  Aircraft. 

Mean  2.33 
SD  1.25 
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16.  I  am  expert  in  the  procedures  required  for  Perform  Degraded 
Mode  Engagement  Against  Hostile  Aircraft. 

Mean  1 . 67 
SD  .47 

17.  I  am  expert  in  the  procedures  required  to  perform  Hangfire 
or  Misfire  procedure. 

Mean  1.00 
SD  .00 

18.  I  am  able  to  recognize  and  evaluate  the  abbreviated  Rearm 
procedure . 

Mean  2.00 
SD  .  82 

19.  I  can  define  and  explain  crew  performance  error  code 
"Procedure  omitted" . 

Mean  1.00 
SD  .00 

20.  I  can  define  and  explain  the  crew  performance  error  code 
"Procedure  performed  inaccurately”. 

Mean  1.33 
SD  .47 

21.  I  can  define  and  explain  the  crew  performance  error  code 
"Procedure  performed  at  the  wrong  time". 

Mean  1.33 
SD  .47 

22.  I  can  define  and  explain  the  crew  performance  error  code 
"Wrong  procedure  performed" . 

Mean  1.33 
SD  .47 

23.  I  can  define  and  explain  the  crew  performance  error  code 
"Extra  procedure  performed". 

Mean  1.33 
SD  .47 
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24.  I  can  define  and  explain  the  crew  performance  error  code 
"Procedure  delay". 

Mean  1.67 
SD  .94 

25.  I  am  able  to  use  all  the  data  provided  by  multiple  sources 
to  determine  the  focus  for  recognizing  operator  interface 
errors . 

Mean  1.67 
SD  .47 

26.  I  am  familiar  with  applying  OTEA  priority  codes  to  crew 
error. 

Mean  2.00 
SD  .82 

27.  I  am  able  to  suggest  or  recommend  corrections  to  existing 
tactics,  techniques  and  procedures. 

Mean  1.67 
SD  .47 

The  following  questions  were  included  to  evaluate  other  aspects 
of  the  training.  They  are  summarized  below  and  student  comments 
are  included. 

28.  Do  you  feel  that  sufficient  time  was  allowed  for  your 
training  as  a  Battle  Drill  Analyst  prior  to  FDTE  II? 

Yes  3  No  0 

29.  What  training,  if  any,  have  you  received  in  the  past  that 
made  this  learning  experience  easier  for  you? 

LOS-F-H;  Various  systems;  NDICE,  MANPRINT  video  reduction; 
PMS  data  collection  and  video  reduction 

30.  Based  on  your  understanding  of  what  is  to  be  tested  at  the 
Fire  Unit  level,  how  many  people  would  you  estimate  are 
needed  to  perform  your  functions  for  each  fire  unit? 


#1 

2 

people 

#2 

2 

people 

#3 

1 

person 
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31.  Please  rate  your  confidence  in  your  ability  to  perform  your 
duties  as  a  battle  drill  analyst  based  on  the  training  you 
have  received. 

Mean  1.33 
SD  .47 

Student  Comments: 

This  training  depends  on  the  availability  of  video  tapes  of 
all  of  the  battle  drills.  These  tapes  were  not  available  or 
were  of  such  poor  quality  that  it  impeded  the  training. 

32.  Please  add  any  additional  comments  which  will  assist  us  in 
improving  and  upgrading  this  training. 

This  training  would  be  greatly  enhanced  by  all  the  players 
participating . 

The  engagement  forms  that  Battle  Drill  Analysis  depends  on, 
the  track  match  to  determine  A/C  available  for  engagement, 
and  the  PCM  data  need  to  be  available.  We  were  able  to 
participate  in  only  some  video  training  without  seeing  the 
entire  process  in  action. 
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